feat: Add resource-info-fetcher by sbernauer · Pull Request #863 · stackabletech/opa-operator

sbernauer · 2026-07-21T08:49:34Z

Description

Part of #848

Yes, the diff looks big, but most of it are lockfiles and kuttl tests

CRD change

Basically the same as for the user-info-fetcher. The only difference is the env field, which is DataHub specific.

Look at extra/crds.yaml for the concrete CRD change

  clusterConfig:
    resourceInfo: # <1>
      backend:
        dataHub:
          hostname: datahub-gms.my-namespace.svc.cluster.local
          port: 8080
          rootPath: / # optional, defaults to /
          tls:
            verification:
              server:
                caCert:
                  secretClass: tls # <2>
          credentialsSecretName: resource-info-fetcher-credentials # <3>
          env: PROD # <4>
      cache: # optional, enabled by default
        entryTimeToLive: 60s # optional, defaults to 60s
---
apiVersion: v1
kind: Secret
metadata:
  name: resource-info-fetcher-credentials
stringData:
  token: <datahub-personal-access-token> # <3>

API of RIF

The rego rules and HTTP API of RIF: https://github.com/stackabletech/opa-operator/blob/spike/rif/docs/modules/opa/pages/usage-guide/resource-info-fetcher.adoc

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

Author

Changes are OpenShift compatible
CRD changes approved
CRD documentation for all fields, following the style guide.
Helm chart can be installed and deployed operator works
Integration tests passed (for non trivial changes)
Changes need to be "offline" compatible
Links to generated (nightly) docs added
Release note snippet added

Reviewer

Code contains useful comments
Code contains useful logging statements
(Integration-)Test cases added
Documentation added or updated. Follows the style guide.
Changelog updated
Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Feature Tracker has been updated
Proper release label has been added
Links to generated (nightly) docs added
Release note snippet added
Add type/deprecation label & add to the deprecation schedule
Add type/experimental label & add to the experimental features tracker

Release notes

Added

Add an initial version of resource-info-fetcher, which is similar to user-info-fetcher, but allows to fetch additional metadata about resource information from a data catalog.
For now only DataHub is supported.
Also, a rego-rule library has been added to make it easier to call resource-info-fetcher from within OPA.
The API (especially the response) might change in the future once more data catalogs are supported

Nightly docs

As soon as this PR is merged the docs will show up at https://docs.stackable.tech/home/nightly/opa/usage-guide/resource-info-fetcher/

lfrancke · 2026-07-30T15:03:29Z

Because we had it yesterday with OpenLineage: I believe it could make sense to add a "path" or "endpoint" field. Just in case DataHub is deployed behind some proxy. ://

sbernauer · 2026-07-31T07:31:35Z

I believe it could make sense to add a "path" or "endpoint" field

Good point! For consistency reasons with UIF and AuthenticationClass I went with rootPath in 215d4b9. Feel free to have a look

Maleware

Just two typos, thanks you! Test passes locally, thus LTGTM otherwise.

Maleware · 2026-08-04T17:14:19Z

+}
+
+/// Generates the trivial `From<Params> for ResourceInfoRequest` conversions, so each HTTP handler
+/// can turn its deserialized query parameters into a [`ResourceInfoRequest`] via `.into()`. Adding a


Suggested change

/// can turn its deserialized query parameters into a [`ResourceInfoRequest`] via `.into()`. Adding a

/// can turn its deserialized query parameters into a [`ResourceInfoRequest`] via `.from()`. Adding a

https://doc.rust-lang.org/std/convert/trait.From.html
From automatically implements Into, so the caller can pick.
In

opa-operator/rust/resource-info-fetcher/src/main.rs

Line 185 in 970da87

P: DeserializeOwned + Into<ResourceInfoRequest>,

we actually use Into<ResourceInfoRequest>. In general I feel in this situation a into feels more natural than a from

Co-authored-by: Maximilian Wittich <56642549+Maleware@users.noreply.github.com>

sbernauer · 2026-08-05T05:59:27Z

Please vote on the CRD change as well as the rego rule API on this comment. Feel free to also look at the JSON API, but I'd say that's an implementation detail we are allowed to change in the future

maltesander

First round, did not check docs etc. properly yet and not testing yet.

maltesander · 2026-08-05T07:43:17Z

+    let resource_info_cache = {
+        Cache::builder()
+            .name("resource-info")
+            .time_to_live(*config.cache.entry_time_to_live)
+            .build()
+    };


The sidecar has 128MB memory and this cache can grow forever. UIF is limited/keyed on usernames from a directory, so this is limited. Set a max capacity (or make it configurable)?

I copied this from UIF and was also surprised there is no cache limit.
However, I disagree that UIF is really different, as the number of users in Keycloak (I guess) is similarly unlimited than the number of resources in DataHub.
So I would be in favor of adding the limit to both caches.
Did so in f481d44

True and +1 for UIF fix. RIF can just grow much quicker in size per request, but yeah the problem is there for both. Thanks!

maltesander · 2026-08-05T07:45:45Z

+        // Trim trailing whitespace/newlines so the value is safe to use in an HTTP header.
+        let token = tokio::fs::read_to_string(&token_path)


This is never refreshed? Will go stale and produce 401 until restart?

Yes, but ResolvedKeycloakBackend::resolve, ResolvedEntraBackend::resolve, ResolvedOpenLdapBackend::resolve` and maybe others do the exact same thing.
According to https://docs.datahub.com/docs/authentication/personal-access-tokens you can create a PAT without expiration.
I see that is not 100% ideal, but also IMHO the status quo basically everywhere else in the platform as well

maltesander · 2026-08-05T07:53:29Z

+fragment ResourceInfo on Entity {
+  # DataHub has no direct "dataProduct" field on assets; membership is a graph edge that points from
+  # the data product to its assets. From the asset's side it is therefore an INCOMING relationship.
+  dataProducts: relationships(input: {types: ["DataProductContains"], direction: INCOMING, count: 10}) {


What if there is something belonging to 11 data products? Why limit to 10? This will hard fail policies that should succeed. Can we paginate that or warn (if a much higher limit than 10 (e.g. 1000)) is reached somehow?

That is a real problem I also noticed but forgot about. It should never happen in reality, so I added a snafu error for it, so it loudly fails and does not silently influence authorization: cd9a38e

maltesander · 2026-08-05T08:16:35Z

+///
+/// [`urn_for_request`]: crate::backend::data_hub::resource_to_urn_mapping::urn_for_request
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub enum ResourceInfoRequest {


The instance is always required. Datahub (defaults) leave that out and a trino table will become e.g. urn:li:dataset:(urn:li:dataPlatform:trino,tpch.sf1.customer,PROD) and the tableResourceInfo always builds {instance}.tpch.sf1.customer which will be empty and therefore fail a correct rego?

Can we make instance optional and skip?

I was thinking of this as well (I actually started without the instance).
I think we should very strongly recommend users to set the instance when ingesting data into DataHub.
Also we likely want to support other backends in the future.
If they don't have such a strange concept of "there is only one Trino" they will require us to send a instance for every request.
So I fear we would have a problem with future backends if we make it optional.

I also had to refactor the crd mod to be only pulled in via lib.rs to fix clippy warnings

sbernauer added 28 commits July 16, 2026 10:18

Initial development of resource-info-fetcher

ccadaf8

feat: Implement Trino catalog and schema

a1da9db

feat: Cache tag and user lookups

d56a198

feat: Lookup Superset things (by id)

f28fdb7

chore: More parallism

90b010d

feat: Add Kafka topic support

53e83c6

Crying in GraphQL

0e2857a

feat: Also fetch domains and data products

551c2f6

RawDataHubUrn -> DataHubUrn

a8310f6

refactor: Move user-info-fetcher sidecar creation into separate module

a25da2b

Merge remote-tracking branch 'origin/main' into spike/rif

e5d1f05

Merge branch 'refactor/uif-sidecar' into spike/rif

9698dc8

Re-add Tiltfile

f5b5bf9

Implement RIF into operator

2cfd78d

Improve test

86127a6

chore: Switch endpoints from complex enum to path based

554a7f2

chore: Move env out of global config to the requests

e5ff314

WIP for Maxi :)

970da87

fix kuttl test

2e4655f

refactor Python assert script

8c9d8c5

refactor Python script

dff0f99

Merge remote-tracking branch 'origin/main' into spike/rif

74c648d

Remove leftover files

32cd6e2

cargo fmt

9d95f1f

Fix OpenShift on high disk usage

d52110c

Merge remote-tracking branch 'origin/main' into spike/rif

26f51e9

cargo update

7c11523

Slightly improve docs

fc889ca

sbernauer marked this pull request as ready for review July 30, 2026 13:49

Stupid YAML linter

59af952

feat: Add rootPath field for DataHub

215d4b9

changelog

4dc05fc

sbernauer self-assigned this Jul 31, 2026

sbernauer added 2 commits July 31, 2026 09:49

Improve code comments

6571afa

regenerate nix

f770b04

sbernauer requested a review from Maleware July 31, 2026 08:53

sbernauer added this to Stackable Engineering Jul 31, 2026

sbernauer moved this to Development: Waiting for Review in Stackable Engineering Jul 31, 2026

Maleware reviewed Aug 4, 2026

View reviewed changes

Update rust/resource-info-fetcher/src/main.rs

8c2a0df

Co-authored-by: Maximilian Wittich <56642549+Maleware@users.noreply.github.com>

sbernauer requested a review from Maleware August 5, 2026 05:56

Merge branch 'main' into spike/rif

af16a08

maltesander self-requested a review August 5, 2026 06:26

maltesander requested changes Aug 5, 2026

View reviewed changes

sbernauer added 2 commits August 6, 2026 16:26

docs: Link in nav

bf85e53

WIP: Error when too many data products are returned

cd9a38e

sbernauer moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Aug 10, 2026

sbernauer added 5 commits August 10, 2026 13:22

Merge remote-tracking branch 'origin/main' into spike/rif

fde8854

fix: Use utils::http::client_builder helper

c20d1d1

clippy

e3e84ed

feat: Allow configuring max cache size.

f481d44

I also had to refactor the crd mod to be only pulled in via lib.rs to fix clippy warnings

regenerate nix

fdcb012

sbernauer requested a review from maltesander August 10, 2026 12:56

	/// can turn its deserialized query parameters into a [`ResourceInfoRequest`] via `.into()`. Adding a
	/// can turn its deserialized query parameters into a [`ResourceInfoRequest`] via `.from()`. Adding a

		// Trim trailing whitespace/newlines so the value is safe to use in an HTTP header.
		let token = tokio::fs::read_to_string(&token_path)

Uh oh!

Uh oh!

Conversation

sbernauer commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

CRD change

API of RIF

Definition of Done Checklist

Author

Reviewer

Acceptance

Release notes

Added

Nightly docs

Uh oh!

lfrancke commented Jul 30, 2026

Uh oh!

sbernauer commented Jul 31, 2026

Uh oh!

Maleware left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sbernauer commented Aug 5, 2026

Uh oh!

maltesander left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maltesander Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sbernauer commented Jul 21, 2026 •

edited

Loading

maltesander Aug 5, 2026 •

edited

Loading